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Abstract 

Lecture videos have become an increasingly prevalent and important source of learning content. Lecturer¬ 
generated summaries may be used during a video lecture to improve student recall. Furthermore, the 
integration of a guest lecturer into the classroom may be a beneficial educational practice drawing the 
learner’s attention to specific content or providing a change of pace. The current study measures the 
effects of lecturer-generated summaries and the inclusion of a guest lecturer on students’ ability to recall 
online video lecture contents. Seven sections of a flipped scientific writing course were divided into three 
groups. The control group videos featured a lecturer speaking with PowerPoint slides in the background. 
The Summaries Only group viewed the same videos as those of the control, with the addition of lecturer¬ 
generated summaries spliced into the middles and ends of the videos, respectively, and these summaries 
were delivered by the same lecturers of the original video. The Summaries with a Guest Lecturer group 
viewed the same videos as the control, but with the addition of lecturer-generated summaries respectively 
spliced into the middles and ends of the videos, and these summaries were instead delivered by a guest 
lecturer. Student recall was measured through two online multiple-choice quizzes. The results of the study 
show that the Summaries Only group significantly outperformed the other two groups, while no 
significant difference was found between the performances of the control and the Summaries with a Guest 
Lecturer group. The results suggest that lecturer-generated summaries help to improve student recall of 
online video lecture contents. However, the introduction of a guest lecturer shown in a different setting 
may cause learners to lose concentration, nullifying the benefit of the summaries. 
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Introduction 

Flipped classrooms are becoming an increasingly prevalent part of the educational landscape (Breslow et 
al., 2013). Since flipped courses tend to rely heavily on online video lectures, it is necessary to examine 
how such videos can affect students’ understanding and recall of course content. While online video 
lectures provide students with considerable advantages in terms of accessibility, and the ability to tailor 
learning to their own pace, there are also valid concerns about the effectiveness of this form of instruction. 
Some major concerns are that students may not be fully engaged in listening to the lecturers or may skip 
them altogether (Alksne, 2016), that students may experience a reduced sense of engagement compared 
to traditional classroom environments, and that video lectures may provide less pedagogical benefit than 
traditional lectures. In e-learning environments, videos and other media can help by keeping learners 
interested and engaged in the material (Zhang, Zhou, Briggs, & Nunamaker, 2006). 

In flipped courses, lectures are typically provided online in video format rather than live in-class delivery. 
While classroom practices may vary, practitioners often recommend that class time traditionally devoted 
to in-class lectures should instead be devoted to more student-centered activities, particularly student- 
centered problem-solving activities (Ferreri & O’Connor, 2013; Kim et al., 2014; Mason, Shuman, & Cook, 
2013; Prober & Khan, 2013; Wilson, 2013). These in-class group activities often require prior knowledge, 
which students are expected to gain from watching online lecture videos beforehand. In such cases, 
student engagement with course videos is important, not only from the perspective of gaining knowledge 
from the videos themselves, but also to maximize the learning opportunities presented through in-class 
collaborative problem-solving activities requiring prior knowledge from the videos. 

There are a broad variety of common methods that are used to improve student retention from lectures, 
both video and traditional formats, including spoken organizational cues (Titsworth & Kiewra, 2004) and 
visual ancillary cues (Hirsch, 1987). Brecht (2012, p. 245) found that lecture videos that had “strong 
presentation of relief and change-of-pace elements” such as alterations in audio or video stimulus, 
background or setting modifications, changes in the camera angle, or dramatic transitions between 
various parts of the video were more effective. Barker and Benest (1996) note that the addition of 
multimedia contents and animations is desirable since it prevents students from losing concentration. 
Furthermore, Costley and Lange (2017) showed that video lectures with a greater variety of media types 
lead to students retaining more of the video contents. Other studies suggest periodically changing the 
pace of a lecture, for example, with a humorous remark, an activity, a question that must be answered, or 
audio/visual material (Padget & Yoder, 2008; Woodring & Woodring, 2011; Wolff, Wagner, Poznanski, 
Schiller, & Santen, 2015; Jones, Peters, & Shields, 2007). A drastic change-of-pace element that can be 
introduced into a video lecture is the appearance of a guest lecturer, which constitutes a type of team 
teaching. The present study will attempt to assess the effect of integrating a guest lecturer into the context 
of video lecture content within a flipped learning environment. 


The level of learning in online environments is another important factor in understanding the 
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effectiveness in video lectures. Quizzes are a commonly-used component across all types of courses and 
are especially prevalent in flipped classroom environments, as quizzes that count towards final grades can 
be powerful motivators for students to actively watch video lecture contents in such learning 
environments (Frydenberg, 2013; Tune, Sturek, & Basile, 2013; Enfield, 2013). Quizzes have often been 
shown to be a good way to measure and motivate student learning (Johnson & Kiviniem, 2009; Herold, 
Lynch, Ramnath, & Ramanathan, 2012; Kamuche, 2011), and in predominantly or entirely online 
environments, such as MOOCs, online quizzes may be the only way to do so easily. In most courses, 
quizzes are connected to the contents of lectures and are seen as a measure of student learning of the 
course content and of lectures. Quizzes can, therefore, be seen as a representation of the amount that 
students learn and retain from the content. Video lectures have been shown to positively impact student 
performance on quizzes (Mason, Shuman, & Cook, 2013; Williams, Birch, & Hancock, 2012), although the 
effectiveness of these videos when compared to traditional lectures is an ongoing debate. 


Literature Review 

The Effects of Summaries on Student Learning 

In order to increase student retention of lecture video contents, various methods are employed by both 
students and instructors. Smidt & Hegelheimer (2004) have shown that among the numerous strategies 
used by students when watching online videos, one of the most common and effective is to listen to the 
content again. A method that instructors use to improve student recall of lecture content is to provide 
cues when important information is being delivered. Instructors have many different means to deliver 
such cues to the students. For example, the lecturer can write important ideas on the blackboard (Locke, 
1977), present it visually (Baker & Lombardi, 1985), or simply emphasize it through the manner of 
speaking (Maddox & Hoole, 1975; Scerbo, Warm, Dember, & Grasha, 1997). Titsworth & Kiewra (2004) 
found that spoken cues increased student academic performance. Each of these methods has been shown 
to bring about an improvement in students’ achievement. Repetition has been shown to improve recall in 
learning situations. Mayer (1983) also claimed that recall and problem-solving abilities increased the 
more that learning content was repeated. Bromage and Mayer (1986) found that repetition enabled 
students to recall more of what they heard and to remember a larger quantity of structurally and 
functionally important information, signaling a change in their learning strategies. Webb (2007) 
examined the effects of repetition on the language acquisition of Japanese students learning English and 
discovered that knowledge increased with increased repetition, as students who encountered a new word 
in context frequently were likely to have a deeper understanding of the word’s meaning and function. 

When providing students with repetition of important content in a course, instructors must also consider 
the pacing of such repetition. An exhaustive number of experiments conducted by cognitive psychologists 
have shown the benefits of spaced repetition of content compared to massed presentation (Cepeda, 
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Pashler, Vul, Wixted, & Rohrer, 2006). Furthermore, distributed practice received one of the highest 
utility ratings in a comprehensive review of a variety of learning strategies based on the available evidence 
(Dunlosky, Rawson, Marsh, Nathan, & Willingham, 2013). It has also been found that providing students 
with a paraphrased version of the content produced an equivalent amount of recall as with a verbatim 
repetition (Glover & Corkill, 1987). This paraphrasing can be seen, in some sense, as a summary of the 
previous content. 

The Effects of Co-Teaching on Student Learning 

A common approach to the structuring and delivery of course material is to divide responsibilities among 
more than one teacher, and several terms are used in the literature to describe this concept, including 
cooperative teaching, team teaching, and co-teaching (Welch, Brownell, & Sheridan, 1999). The definition 
of co-teaching according to Cook and Friend (1996) is “two or more professionals delivering substantive 
instruction to a diverse, or blended group of students in a single space” (p. 156). Cook and Friend (1996) 
describe the following five styles of co-teaching. The first of these is called “one teaching / one assisting,” 
which means that one instructor takes primary responsibility for the delivery of content, while the other 
plays a facilitatory role. One issue with this method is that the assisting teacher may be seen more as an 
assistant than an equal partner in instruction. “Station teaching” refers to teachers dividing presentation 
of the course content as well as the physical space of the learning environment, with each teacher 
managing a separate part of the classroom and the curriculum, with student groups alternating between 
different segments of the classroom. “Parallel teaching” is where teachers collaborate in planning course 
content but then divide the students into groups of equal sizes and are responsible for educating their own 
respective groups. “Alternative teaching” usually involves dividing a class into one large and one small 
group, with the smaller group receiving specialized instruction such as pre-teaching, directed practice, 
and review. In “team teaching,” the instructors take turns in discussing course topics with students or in 
providing demonstrations through role play. As an alternative to co-teaching, an instructor may choose to 
bring in a guest lecturer, as the guest lecturer may have more expertise on a particular topic, or in some 
cases, provide students with a change of pace or a differing perspective. This can also be done in online 
videos, with cameos by guest lecturers. 

It is often claimed that students’ attention spans fall within the 10- to 20-minute mark (Davis, 1993). 
While the concreteness of this time-frame is debated (Wilson & Korn, 2007), the occurrence of such 
lapses is not (Bunce, Flens, & Neiles, 2010). This phenomenon is often called the “vigilance decrement” in 
the literature (Ariga & Lleras, 2011; Farley, Risko, & Kingstone, 2013; Risko, Anderson, Sarwal, 
Engelhardt, & Kingstone, 2012; Young, Robinson, & Alberts, 2009). To counter the negative effects that 
attention lapses may have on learning, switching things up or changing pace is recommended to ensure 
students remain focused (Center for Excellence in Teaching, 1999). One method of doing so includes 
introducing a guest lecturer to maintain student interest in the lesson (Young, Robinson, & Alberts, 2009). 
While the literature on team teaching often refers to traditional rather than online or blended learning 
environments, studies have shown that a variety of student benefits to team teaching, including exposure 
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to a variety of viewpoints (Letterman & Dugan, 2004), improved achievement and retention of course 
contents (Johnson, Johnson, & Smith, 2000), improved communication skills (Helms, Alvis, & Willis,, 
2005) and better student-teacher relationships (Wilson & Martin, 1998). Interestingly, when Dugan and 
Letterman (2008) compared student’s own perceptions of the value of team-taught courses compared to 
traditional courses taught by one instructor, they found no significant difference between them. Despite 
this finding, Dugan and Letterman did find that students preferred courses taught by two instructors 
rather than a panel of three or more instructors, and that the students preferred co-teaching, where each 
instructor attends and actively engages in teaching the class during each session, over an alternating style 
of instruction where each instructor would teach during an entire class session on an turn-by-turn basis. 

The Effects of Quizzes on Student Learning 

Quizzes have been shown to increase the long-term retention of academic lecture content as compared to 
restudying or not reviewing the contents (Roediger III & Karpicke, 2006; Butler & Roediger III, 2007). 
Marks (2015) points out that computer-based quizzes enable instructors to pinpoint if some or all 
students have been struggling with the lesson’s content, allowing the lecturer to adjust instruction 
accordingly. For these reasons, quizzes can be a valuable component in flipped and blended classrooms 
where students are expected to perform computer-based tasks outside of the class and to engage in group 
work during class time. 

Multiple-choice quizzes are pervasive in university-level courses (Roediger & Marsh, 2005). While they 
have sometimes been disparaged in the literature, multiple-choice quizzes are often viewed as a necessary 
evil since they provide instructors with an easy way to grade a large number of quizzes, particularly for 
large undergraduate courses, where short answer or essay exams may be too burdensome or unreliable 
(Little, Bjork, Bjork, & Angello, 2012). For the same reasons, multiple-choice quizzes are widely used in 
MOOCs and other e-learning environments in which large numbers of students are enrolled. (Colvin et al., 
2014). Commonly cited criticism of multiple-choice quizzes include their limited ability to measure 
complex learning (Frederiksen, 1984), their inability to engage students in the type of retrieval processes 
that lead to long-term retention (Chan, McDermott, & Roediger, 2006; Foos & Fisher, 1988), and their 
tendency to cause misinformation since studies have shown that simply exposing students to plausible 
incorrect answers causes them to judge such lures as more true than novel fact answers, though never 
quite reaching the rated truth levels of true statements (Roediger & Marsh, 2005; Toppino & Brochin, 
1989; Toppino & Luipersbeck, 1993). 

However, studies have also shown advantages to multiple-choice testing. For example, despite the 
aforementioned issue of the creation of false knowledge, Roediger and Marsh (2005) showed a substantial 
positive testing effect when students were given a multiple-choice test in preparation for a general 
knowledge exam. Little et al. (2012) found that properly constructed multiple-choice questions fostered 
the recall of previously tested information, where “proper construction” means that all answer choices are 
plausible, but not so plausible that the question becomes unfair. They further found that multiple-choice 
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questions aided students in the recall of information related to incorrect answer choices, unlike cued- 
recall tests (Little et al., 2012). Multiple-choice questions also offer greater reliability in grading and 
provide students with a greater variety of questions in a shorter period of time than some other 
constructed response questions, such as essay exams (Walstad & Becker, 1994). A common criticism of 
multiple-choice questions is that testing factual knowledge does not guarantee competence, as high-level 
competence requires the integration of knowledge with attitudes and communication skills (McCoubrie, 
2004). However, research has shown that knowledge of a subject area is the single best determinant of 
expertise (Glaser, 1984). Therefore, multiple-choice questions are a valid method of testing competence, 
as written test forms are the best assessment instrument of cognitive knowledge (Downing, 2002). 

Relationship Between Summaries and Quizzes 

In a study by King (1992), university students were divided into three groups, one trained in self¬ 
questioning (and answering these questions), one trained in summarizing, and one untrained control 
group who simply took notes and reviewed them. Among the three groups, King found that the summary 
and self-questioning groups outperformed the control group in the recall of a lecture, with the summary 
group performing best in the short-term and the self-questioning group performing slightly better than 
the summary group in the long-term. 

Relationship Between Co-Teaching and Quizzes 

Adapting the concepts of co-teaching mentioned previously, Jang (2006) devised an experimental team¬ 
teaching approach in which two teachers specialized in teaching different parts of a science lesson, and 
were transferred to different classrooms at different times to provide instruction on that specialization. 
This approach was combined with a course website where students could access supplemental textbook 
information, discussion groups, and class notices as well as turn in course assignments. Jang found that 
students who were taught by this team-teaching web-based approach showed more improvement from 
their pretest to their posttest scores as compared to students who took traditional classes taught by one 
teacher with no web component (Jang, 2006). 

The Present Study 

Considering the need for developing lectures that are engaging and effective for learners, this study is 
guided by two research questions: 

1) Do video lectures with summaries result in higher scores for students on related online quizzes? 

2) Do video lectures with a guest lecturer result in higher scores for students on related online 
quizzes? 
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Methods 


Experimental Procedures 

In the present study, we sought to assess how change-of-pace and structural elements in online videos 
would affect student learning outcomes in a Scientific Writing course (CC500) at the Korea Advanced 
Institute of Science and Technology (KAIST) in Daejeon, South Korea. This Scientific Writing course is 
specialized compared to other academic writing courses offered at KAIST in that it is only offered for 
graduate students, and it focuses on producing and publishing research manuscripts in science and 
technology. This focus means that the video lectures cover numerous topics beyond traditional writing 
skills and include issues such as the function of particular sections of a manuscript and various 
conventions of scientific and engineering writing, as well as the process of publishing a paper. The course 
is offered in a flipped format, so that students and professors met once per week, and students also 
watched videos and took quizzes online for homework. 

The present study examined 3 weeks (11 videos) among the total 16 weeks (56 videos) that constitute the 
entire course. The first batch of five videos were shown to the students in week 6 of the 16-week course 
and covered the following five topics: “Usage Rules for the Colon and Semicolon,” “Usage Rules for the 
Hyphen,” “How to Incorporate Numbers into Writing,” “Avoiding Unclear Pronoun Usage,” and 
“Paraphrasing.” The second batch of six videos was shown in weeks 14 and 15 of the 16-week course and 
all applied to the theme of “Publishing Your Paper.” This video series was designed to guide the students 
through the steps that follow the completion of a research manuscript, and the topics were as follows: 
“Selecting a Journal,” “Open Access vs. Subscription Journals,” “How to Get your Research Published,” 
“The Peer Review Process,” “Common Reasons for Rejection,” and “Final Thoughts and Advice on 
Publishing Your Paper.” 

For the control group, the video series was prepared in a straightforward style with only one lecturer and 
no summaries, as shown in Figure 1. For the videos prepared for the "summaries only" group, all lecturing 
and summaries were delivered by the same instructor, as shown in Figure 2(a). The summaries were 
around 30 seconds in length and were shown at the middle and end of each video. In the mid- and end¬ 
point summaries, the lecturer would mention the main points from the first and second halves of each 
video, respectively. The video series provided to students in the "summaries with a guest lecturer" group 
contained more varied change-of-pace elements: each video contained mid and final summaries with 
identical content to those of the "summaries only" group, but the summaries were delivered by a guest 
lecturer who was also seated in a “coffee shop” setting projected in the background, as shown in Figure 
2(b), rather than standing in front of PowerPoint slide contents, as in the videos of the "summaries only" 
group. 
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Figure l . Main video lecture without summaries. 



Figure 2. (From left to right) Summaries by the main lecturer (a) and guest lecturer (b). 


Control group video timeline (no summaries) 

(A) 

Treatment group 1 video timeline (summaries by main lecturer) 

(A)-(B)-(A)-(B)-(A)- 

Treatment group 2 video timeline (summaries by guest lecturer) 

(A)-(C)-(A)-(C)-(A) --- 

Figure 3. Summary of the video timelines. The letters A-C correspond to the style of videos shown above. 
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A total of 135 students were divided up into a control group and two treatment groups, with between 39 
and 55 students in each group. The respective videos were posted on the school’s learning management 
system in weeks 6 and 15 and were available to watch at the students’ convenience. Once the videos were 
viewed for each week of the study, students took a multiple-choice quiz online, which they could access at 
any time during the seven-day video viewing period but which must be completed by the respective brick- 
and-mortar classroom meeting day. The quiz was used to measure the students’ comprehension and recall 
of the contents of the videos. During the respective brick-and-mortar classroom meeting, students were 
asked to fill out a survey that involved a 10-point Likert scale to assess the videos. Survey forms were 
assessed, and quiz data was taken from the online learning management system of the course. 

Participants and Context 

This study was conducted at the Korea Advanced Institute of Science and Technology (KAIST), a large 
university located in Daejeon, South Korea. Most students at the university specialize in STEM fields. As 
of 2013, KAIST has a student population of 11,175, with about 60% of those studying at the graduate level 
(KAIST, 2014b). The university is predominantly male (80%), and international students make up 5% of 
the total population (KAIST, 2014b). KAIST offers an array of online and blended courses. These include 
Massive Open Online Courses (MOOCs) made available through Coursera, institution-level online courses 
through the CyberKAIST program as well as the Bridge-Program for prospective freshmen, and global and 
institutional-level flipped courses through iPodia and Education 3.0, respectively (KAIST, 2014a). 

Participants in the present study were students in classes that were provided as part of KAIST’s Education 
3.0. The Education 3.0 initiative was started in 2012 with the goal of reducing the amount of traditional 
lecturing in KAIST courses and enabling students to engage in more interactive and communicative 
learning activities through a flipped classroom environment (Horn, 2014). KAIST performed a trial run 
involving four Education 3.0 courses in 2012. Students’ performance on exams and overall satisfaction 
with the courses were higher than those with conventional lecture classes (Horn, 2014). In 2014, 5% of all 
classes at KAIST were offered in Education 3.0 format, with a goal of increasing that number to 30% by 
2018 (Horn, 2014). As a graduation requirement, master’s and PhD students at KAIST must be able to 
compose articles for publication in scientific and engineering journals. To assist them in this process, the 
Scientific Writing (CC500) course teaches students how to communicate their research through writing in 
English. The course is conducted in English, and enrollment is generally 20 students per section. In the 
present study, seven sections of Scientific Writing were included in the experiment. This was a total of 135 
participants, 17 of which did not complete the second section of the quiz and so were removed from the 
analysis done as a part of this study. This left 118 valid responses, of which 32 were female and 86 were 
male. The oldest participant was 45, and the youngest was 22, with a mean age of 27. The seven selected 
sections of Scientific Writing were taught as part of KAIST’s Education 3.0 program, so they were given in 
a flipped format. 

Measures 
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Students took two quizzes, the first comprising 25 questions and covering information given in five video 
lectures and the second comprising 30 questions covering six video lectures. All quizzes were given online 
and could be taken at any point during the one-week video viewing period (weeks 6 and 15, respectively). 
The quizzes were made up of multiple choice questions, with some of the quizzes allowing only one 
answer choice and others calling for one or more answer choices, and example of some of the questions 
can be seen below in Table 1. For the latter question type, partial credit was awarded when a correct 
answer was selected, but no credit was given when an incorrect answer choice was selected. Quiz 
questions were created by the course instructors and were designed to make students demonstrate that 
they could apply the concepts taught in the videos. Each quiz was worth 5% of the class’s total grade. 
While the effectiveness of multiple-choice quizzes is still being debated, they were an attractive choice for 
this course, as with most e-learning environments, since they present an opportunity to measure students’ 
recall while allowing students to take quizzes anytime and anywhere (provided they have internet access 
and a device) with immediate feedback. The topics in the first batch of videos were related to rule-based 
concepts rather than communicative issues in writing, and provided descriptions and suggestions to guide 
students through the process of publishing their completed manuscripts in a journal after the conclusion 
of the course. We felt that such topics were appropriate for measurement via quizzes, while the quality of 
the students’ writing and their ability to clearly communicate their research results were evaluated 
separately through instructor comments and grading. 

Table 1 

Sample Quiz Questions 


Quiz and 
question 

numbers Question Choice of answers 


Quiz 3, 
Question 1 


Which of the following 

sentences is correct in terms of a . 2 apples, three oranges, and five bananas were 
writing with numbers? Select enough fruit for the class, 
one or more answers. 


b. According to the survey, people generally slept 
seven point five hours per night. 


c. The club celebrated the birthdays of the five 
90-year-olds who were born in the city. 

d. 6% of the class failed. 
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Quiz 3, 
Question 8 


Quiz 6, 
Question 20 


Quiz 6, 
Question 28 


Which of the following is true 

regarding paraphrasing? Select a. Information can be copied word for word, 
one or more answers. 

b. You use the same words and phrases. 


c. Paraphrase is generally the same length or 
slightly shorter than the original source. 


d. You don't have to mention where the original 
information came from if you paraphrase. 


The aim and scope of a journal 

_. Select one a. should be inferred by reading. 

or more answers. 

b. is less narrow in high impact journals. 


c. can be ignored if the submitted data is highly 
revolutionary. 


d. can be a deciding factor when accepting or 
rejecting a submission. 


e. often change to adapt to trends. 


Which of the following is an 

advantage of a journal transfer? a. It leads to a higher impact publication. 
Select one or more answers. 

b. It saves time for the authors. 
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c. It separates scientific soundness from interest 
level. 

d. It saves time for the reviewers. 


Results 

The first point of analysis was to look at any relationships between the demographic variables of gender 
and age the dependent variable. Using Spearman’s point-bi-serial correlations for gender, and Spearman’s 
standard bivariate correlations for age and quiz score, correlation coefficients were calculated and 
analyzed, which can be seen in Table 2 below. The results showed that the main dependent variable for 
the study (quiz score) had a small non-statistically significant relationship with both age and gender. 
However, there was a statistically significant relationship between age and gender. The negative 
relationship between age and gender was -.199, the relationship between gender and quiz score was .011, 
and age and quiz score was -.051. The likely reason for the relationship between age and gender is that 
Korean males are required to do two years of military service, which is usually done after their first or 
second year of university. These two years of time out of education means that men in graduate school in 
Korea tend to be older than women. 

Table 2 

Correlations Between Gender, Age, and Grade 



Gender 

Age 

Grade 

Gender 

1 



Age 

-.199** 

1 


Quiz score 

.011 

-.051 

1 


** p = <.01 


To answer the main research questions that were a part of this study, the mean quiz scores for each 
experimental group were calculated and compared (see Table 3). The condition with the highest mean 
quiz score was the condition with the summary only, followed by the summaiy with a different professor, 
with the basic lecture having the lowest mean quiz score. It is worth noting that both the basic lecture 
condition (5.789) and the summary with a different professor (5.827) had very similar means, while the 
condition that contained summaries only had markedly higher mean quiz scores (6.787). 
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Table 3 

Mean Quiz Score for Each Experimental Condition 



Mean 

N 

SD 

Min 

Max 

Basic lecture 

5-789 

46 

1.35867 

2-53 

9.00 

With summary 

6.787 

37 

1.68573 

3.01 

10.00 

With summary and different professor 

5.827 

35 

1.44842 

2.99 

9-53 

Total 

6.134 

118 

1-55056 

2-53 

10.00 


After examining the quiz means for each of the three conditions, Analysis of Variance (ANOVA) was used 
to analyze the differences among and between the group means that were a part of this research. ANOVA 
tested if there was a statistically significant difference between the basic lectures, lectures with summary 
only, and lectures with summary by a guest lecturer groups. Table 4 shows that there was a statistically 
significant difference between the three groups that were a part of this study (p = c.oi). 


Table 4 

ANOVA Comparisons for Grades Within the Three Experimental Conditions 



Sum of squares 

df 

Mean square 

F 

Sig- 

Between groups 

15.812 

2 

7.906 

3-633 

•005 

Within groups 

287.250 

132 

2.176 



Total 

303.063 

134 





After the overall difference among the experimental groups was established, the Scheffe test was used to 
examine the specific difference between each experimental group’s quiz means, and whether or not those 
differences were statistically significant. The Scheffe test is a single-step multiple comparison procedure 
designed to be applied to the set of estimates of all possible comparisons among means. Table 5 shows 
that in the lecture with summary group, the quiz mean was .998 higher than that of the basic lecture 
group. Also, the lecture with a summaiy and different professor group had a quiz mean .960 lower than 
the lecture with summary group. Both the relationships between the basic lecture and lecture with 
summary, and lecture with summary with a guest lecturer and lecture with summaiy were statistically 
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significant. The quiz mean of the lecture with summary by a guest lecturer was .038 higher than the basic 
lecture quiz mean, though this relationship is not statistically significant. 

Table 5 

Scheffe Test for Comparisons of the Three Experimental Conditions 



Basic lecture 

With summary 

With summary and 
different professor 

Basic lecture 

1 



With summary 

.998* 

1 


With summary and 
different professor 

.038 

-.960* 

1 


*P = <-05 


A post-hoc survey was conducted to collect students' opinions on their experiences regarding the video 
lecture element of the course once the semester had ended; grades had already been allocated prior to the 
email messages being sent. Said messages contained screenshots of the summaries each respective 
treatment group had encountered to serve as a reminder, as well as a simple request for participants’ 
feelings regarding the summaries; any general comments on the videos were also welcomed. A balanced 
amount of criticisms to positive responses were logged. 

Students from the two treatment groups generally perceived the summaries positively, noting they were 
helpful for reviewing and emphasizing the salient points of the lecture. Such summaries were seen to 
complement notetaking and were appreciated as they enabled a time to reflect and take stock of what had 
been explained so far. For weaker students, they were seen as a tool for clarifying understanding of the 
content. In addition, the appearance of a guest lecturer for those in the “summaries with a guest lecturer” 
group was said to have a “refreshing” function by some participants - as if it were a reminder to focus 
one’s attention. One individual in particular mentioned how useful such appearances were to draw one’s 
attention back to the content if they were multitasking. Others claimed the summaries reduced the 
amount of pausing and (presumably) re-watching. 

It was, however, felt (by members of both treatment groups) that while generally useful, summaries may 
be redundant for shorter videos or those that deal with straightforward content. There was a tendency to 
favor one final summary in such cases. Furthermore, a common complaint for the second treatment group 
was that the summaries were distracting as they broke up the structure and disturbed the flow of the 
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lecture. 

Of the 19 replies received, the following represent the spectrum of responses. 

Negative Responses to the Summaries 

Redundancy. 

“If the content was difficult, full of things to memorize, or a lot of additional explanation, then the 
mid summaries were very helpful. However, if the content is straightforward, I think mid 
summaries might be redundant.” (Ti) 

“Mid summaries were such a time consuming point in the lecture in some moments in that it re¬ 
explained the content which were studied right before (even in few seconds).” (Ti) 

“It could make students distracted. I think a concise lecture with only end summary will be great 
to let student know what is the most important concept on that lecture and what should be 
remembered and studied.” (Ti) 

Distraction. 

“But I think there may not need the time countdown - It distracts my focus a little.” (T2) 

“I did think the coffee scene broke up the structure of the lecture.” (Treatment 2) 

Neutral to Negative Responses to the Summaries 

Students as directors of learning. 

It would be better to collect those summarizing slides and provide them as ppt/pdf. The reason is 
... if students already know some of the subsections of the lecture, they can skip those parts and 
dive directly into the subsections they don't know about. (Ti) 

If I were to think back and give you some recommendations, I think having an option of speeding 
up the videos will be great. I used to watch 3-4 videos in parallel because there was no speeding 
option. Perhaps because I was watching the videos intermingled together, I felt the need to focus 
more during the summary sections, but I think they are useful anyways because I doubt most 
students don’t multi-task or think of what they are having for dinner when listening to the videos. 
(T2) 


Positive Responses to the Summaries 

Help Clarify Content for Weaker Students. 
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Actually, because of some reasons (poor English skill, not concentrating on the subject, etc.), I 
couldn’t understand the words quite many times. But because of the summaries you gave us, I 
could catch up on the lecture and understand what's going on at that time. (Ti) 

Focus Attention (during Attention Lapses). 

“I used to get spaced out sometimes while watching videos, but the summarizing slides help me 
focus on the topics and keep track of the lectures.” (Ti) 

“Appearance of the guest professors refreshed me to focus on the videos. Of coarse [sic], the 
summary itself helped me to memorize key points of the lectures.” (T2) 

“Sometime[s] it is easy to be distracted while watching the video so, summary gives a nice 
construct of the contents discussed.” (T2) 

Focus Attention on the Most Salient Points. 

“When I was attained in Edu3.o online lecture, I usually paused the video to understand unclear 
contents. However, these mid and end summaries reduced that, because these showed the key 
points of the talks.” (T2) 

“I enjoyed having some form of summary as this gave me an idea of the most important points of 
the lecture, giving me a focus on what to revise.” (T2) 

A Rest Point. 

Regarding the mid and end summaries, I think they were actually the best part of the video 
lectures. I appreciated having a stopping point in the lectures, and it was helpful having the 
summarized version in my head before taking the quizzes. (Ti) 

Assisted Note-Taking. 

“Having summaries definitely coincided with my note taking formats and thus was very effective.” 
(Ti) 

“Also, in my case, summary slides were useful for preparing the quiz and making a summary note.” 
(Ti) 

Useful as a Recap. 

In particular, I like mid summaries because I can refresh in the middle of the lecture. Lectures are 
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sometimes long, and sometimes I cannot remember all the contents of the lecture. The mid 
summaries help to understand the flow of content. (Ti) 


Discussion 

The students in the "summaries only" group scored an average of nearly one point higher than their 
counterparts in the control and the "summaries with a guest lecturer" group on the combined to points of 
the two 5-point quizzes given in weeks 6 and 15, respectively. A number of studies in the literature suggest 
that the use of lecturer-generated summaries, whether spoken or visual, increases learner retention of 
course content (Hirsch, 1987; Titsworth & Kiewra, 2004). The improved quiz scores of the "summaries 
only" group as compared to those of the control seem to support this notion. However, the usefulness of 
lecturer-generated summaries seems to be contradicted by the result that there was no significant 
difference between the scores of the "summaries with a guest lecturer" group and the control. A reason to 
expect that summaries with a guest lecturer would have outperformed the other two groups is that several 
studies have shown that the integration of a guest lecturer into a traditional classroom setting can be 
beneficial (Jang, 2006; Johnson et al., 2000; Letterman & Dugan, 2004). We expected this tendency to 
manifest in online course videos, but this did not occur. 

One possible explanation for this result is that some aspect of instructor characteristics caused the lack of 
increase of retention among students who viewed the videos with a guest lecturer. Numerous studies have 
shown that unfamiliar accents may impede comprehension (Eisenstein & Berkowitz, 1981; Smith & 
Bisazza, 1982; Anderson-Hsieh & Koehler, 1988), and students often identify accents as an obstacle in 
understanding a lecture (Richards, 1983; Bilbow, 1989). In the present study, one of the instructors spoke 
standard American English, while the other spoke standard British English. Considering that the students 
also met their respective instructor once a week over a 16-week period during the semester for a workshop 
session in addition to watching the video lectures, it is plausible that the students developed familiarity 
with the accent and speaking mannerisms of their primary instructor. Perhaps the introduction of a guest 
speaker with an unfamiliar accent caused difficulties in listening comprehension. 

A number of studies have recommended and also shown that change-of-pace elements help maintain 
students’ attention and enable them to better recall course content (Brecht, 2012; Center for Excellence in 
Teaching, 1999). Because the videos shown to the "summaries with a guest lecturer" group contained a 
greater variety of presentation elements (e.g., the change of scene to a coffee shop setting, a 30-second 
stopwatch animation, and the presence of a guest lecturer), we expected the videos shown to the 
"summaries with a guest lecturer" group to be the most effective. The sudden change in the setting of the 
video from a familiar-looking PowerPoint slide background to an informal coffee shop scene may have 
seemed jarring to the students, causing an interruption to their concentration on the content of the video. 
One student from the “summaries with a guest lecturer” group described his feelings on the videos as 
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follows: “The summaries cut the flow of the lecture. I was focused on the [content] of the video but the 
change of professor and background looked so complex. It was disturbing [sic].” 

Kim et al. (2014) point out that increases in viewership and student lecture behaviors, such as pausing, 
seeking backwards and forward, and re-watching videos, occur during particularly unclear or interesting 
segments of a video. As evidence that the visual design of a video can affect the ability of students to 
understand the information covered, Kim et al. found, in their study, that 61% of such increased periods 
of viewership and lecture behaviors occurred during an abrupt change in the background, such as shifting 
from a slide view to a classroom view, and the researchers further suggest that video content creators 
refrain from including abrupt scene changes or excessive transitions, as these may cause student to feel 
confused or to lose the context. While we were unaware of this suggestion when designing the treatment 
videos and the present experiment, the aforementioned findings provide a plausible explanation for the 
reduced effectiveness of the videos in the “summaries with a guest lecturer” group compared to the 
“summaries only” group, and our findings seem to support those of Kim et al. (2014). While both of these 
groups contained mid- and end-point summaries of equal length (30 seconds each), only the “summaries 
with a guest lecturer” contained scene changes (from a slide background to a coffee shop setting and back 
again), as the “summaries only” group kept the same slide background throughout. 


Limitations and Future Research 

When considering these results, it is also clear that the present study has a number of limitations. Only 
two weeks of the videos and quizzes were examined, but examining more cases would have provided a 
more complete view of student outcomes. Furthermore, inclusion of a greater number of course sections 
of scientific writing, and thereby a greater number of participants, would have improved reliability of the 
results. 

Another limitation is that we were unable to get a clear sense of the guest lecturer effect from the data 
since the “summaries by a guest lecturer” videos also contained abrupt scene changes. How would 
summaries given by guest lecturers have compared with summaries given by the original lecturer if each 
was provided with a minimal scene change (e.g., keeping the same PowerPoint slide background)? A few 
students who responded to the email survey mentioned that they found the mid-point summaries jarring 
or interrupting to the flow of the lecture. One way to address these concerns would be to provide only end¬ 
point summaries, but the present study did not measure the effectiveness of such an approach. As 
students noted that summaries were useful for emphasizing the most salient points of the lecture (Table 8) 
and in keeping with Guo, Kim, & Rubin (2014) who suggest that video lecture designers should “facilitate 
skimming and re-watching,” we propose that instructors create timeline markers for summaries, so 
students are aware of the presence of something at a clearly marked point (perhaps shown by a dot or 
vertical line) on the video timeline bar. Such a feature will address the aforementioned complaint because, 
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eventually, it will become obvious that the marker indicates the position of the summaries. 

Furthermore, in the present study, the guest lecturer spoke using a different type of English from the 
primary course lecturer (i.e., British vs. American English). Summaries delivered by the primary lecturer 
improved student recall, but those delivered by a guest lecturer who spoke a different type of English did 
not; however, it is unclear whether guest-lecturer summaries were ineffective due to the difference in the 
types of English being spoken between the two lecturers or if the mere presence of a guest lecturer is, in 
and of itself, distracting to student concentration on course content. This question could be answered by 
assessing the effects of lecturer-generated summaries delivered by a guest lecturer who speaks the same 
type of English as the primary course lecturer. 


Conclusion 

While we cannot say with certainty which of the aspects of the videos in the "summaries with a guest 
lecturer" group distracted or hindered students in recalling the contents, it is clear from the results of the 
present study that mid- and end-point summaries provided and delivered by the original instructor 
enabled students to perform better on related quizzes. With this in mind, we recommend that instructors 
incorporate short summaries into course videos in MOOCs and flipped and blended classes. The findings 
of the present study are relevant to instructors in e-learning environments because our data suggest that 
inserting short summaries into existing course videos is a simple and facile way to improve student recall 
of course contents. Even though a number of studies in the literature suggest that a greater number of 
change-of-pace elements will increase student recall of video contents, instructors should also bear in 
mind that, in certain instances and to certain degrees, such elements may become distractions that may 
eventually negate the benefits they are expected to provide. This was particularly evident in the case of the 
scene changes between the slide background and the coffee shop setting in the “guest lecturer summaries” 
group. As suggested by Kim et al. (2014), avoiding abrupt scene changes in video lectures appears to be 
sound advice for video content developers for flipped classrooms, MOOCs, and other e-learning 
environments, and this suggestion was confirmed in the present study. 
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